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Introduction 

The No Child Left Behind Act of 2001 has increased the role of assessment in K-12 
education. Designed to help ensure that all students meet high academic standards, the law 
currently requires states receiving Title I funds to test all children annually in reading and math 
in grades three through eight and report student performance disaggregated by poverty, race and 
ethnicity, disability, and limited English proficiency. By the 2005-06 school year, tests must be 
expanded to include at least one year between grades 10-12, and by 2007-08, states must also 
include science assessments at least once in grades 3-5, grades 6-9, and grades 10-12. The law 
requires states to set annual measurable objectives to track student progress towards reaching 
proficiency, with the ultimate goal that “all groups of students — including low-income students, 
students from major racial and ethnic groups, students with disabilities, and students with limited 
English proficiency — reach proficiency within 12 years” (U.S. Department of Education, 2002, 
p. 17). 

With this goal in mind, school districts are scrambling to develop assessment systems that 
will enable them to monitor student progress in a timely fashion rather than waiting for year-end 
statewide assessments. These district assessments serve multiple purposes: monitoring student 
progress, evaluating the effectiveness of particular programs and schools, and providing school 
personnel with valuable information about how well they and their students are doing. 
Developing easy to administer and score assessments at the district level offers schools a distinct 
advantage over depending on costly statewide assessments for progress monitoring. In the area 
of reading, three measures can provide essential information about students’ developing 
proficiency: a test of oral reading fluency (ORE), a vocabulary test, and a reading comprehension 
test comprised of selected response and constructed response items. Taken together, these three 
measures give a good prediction of student performance on the large-scale reading assessment 




Reading Analysis 7* Grade - Page 2 



administered by the state. To be most useful at the distriet level, it is helpful to have a variety of 
comparable forms available for each of these measures so that students can be tested more than 
once each year without skewing the results due to a practice effect with the same items. 

Methods 

Setting and Subjects 

This report summarizes the spring 2003, seventh-grade reading achievement data from 
five different schools in an urban school district in a mid-sized city in the Pacific Northwest. The 
original data set contained 358 students, but 96 students were removed from the data set prior to 
analysis because they had no scores in any of the dependent variables. Additional students were 
missing data in some but not all of the dependent variable measures, so the total sample size used 
for analysis varies by measure. 

Design and Operational Procedures 

Dependent variables analyzed in this report include scores from the following measures: 
a test of Oral Reading Fluency (ORF) {n = 303), a District Vocabulary Test {n = 327), and a 
District Reading Comprehension Test comprised of both selected response and constructed 
response items {n = 327). All seventh-grade students in the classes selected by the district to be 
sampled present in school on the days the tests were administered took all three assessments. 
Measurement/Instrument Development 
ORF 

The test of Oral Reading Fluency was administered individually to each student by 
trained assessors. Students read aloud for exactly one minute one of four comparable passages 
deemed grade-level appropriate on the Flesch-Kinkaid reading scale. At the end of one minute. 
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assessors marked the last word read then counted the total words read as well as any words read 
incorrectly to arrive at a final ORF score. 

Vocabulary 

Seventh-grade students were administered one of two multiple choice Vocabulary Tests. 
Both tests contained 70 questions. Each item on both forms consisted of one correct answer and 
two distracters. Students bubbled in their answers on the form itself, and all tests were machine 
scored. Differences in student performance on the two forms were not statistically significant 
F(l,300) = .223,p> .05. 

Reading Comprehension 

In addition, seventh-grade students were administered one of four Reading 
Comprehension Tests. Each form of the Reading Comprehension Test consisted of a reading 
passage followed by multiple choice as well as constructed response questions. Multiple choice, 
or selected response (SR), questions were machine scored while constructed response (CR) 
questions were all scored by the same scorer using scoring guides provided by the district. The 
scorer was trained by two district administrators who also checked every fifth paper to ensure 
that his scores were consistent with district expectations. Responses for which the scorer was 
unable to decide on an appropriate score were discussed with both trainers before having a final 
score assigned. 

Data Preparation and Analysis 

I used AOV to test for form comparability and differential performance by different 
groups of students. I then ran an option analysis using Excel, calculating the percentage of 
students selecting each response, the mean score on the measure for the students selecting each 
response, and the correlation between score on the measure and response selection for each item 




Reading Analysis 7* Grade - Page 4 



(see Appendix A). Because the SR section of Form A of the District Reading Test was found to 
be not comparable with the other three forms, data from Form A was excluded in the correlation 
and multiple regression analyses. Alpha level was set at .05 for all analyses. 

Results 



ORF 

Table 1 presents descriptive statistics for the district seventh grade ORF test. 



Table 1 



Descriptive Statistics for Grade 7 District ORF Test 





Group 


n 


M 


SD 


Gender 


Male 


141 


151.96 


33.63 




Female 


146 


159.16 


36.28 


Total 




287 


155.62 


35.13 



There were no significant differences between student performance on the ORF based on gender 
F (1, 285) = 3.04, p > .05 (see Table 2). 

Table 2 



Analysis of Variance Summary Table for Grade 7 District ORF Test 



Source 


df 


F 




P 


Gender 


1 


3.04 


.01 


.08 


Error 


285 


(1225.31) 







Note. Values enclosed in parentheses represent mean square errors. 
*p < .05, **p < .01. 
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District Vocabulary Test 

Table 3 presents a comparison of student performance on Forms A and B of the District 
Vocabulary Test. There was no statistically significant difference between student performance 
on Forms A and B, F (1, 300) = .223 p > .05. However, both forms could yield a more detailed 
picture of differentiated student achievement if they were made more challenging. 

Table 3 



Comparison of Forms A and B of Grade 7 District Vocabulary Test 



Form 


n 


M 


SD 


A 


153 


75.15% 


19.73 


B 


149 


74.16% 


16.64 



Because there was no significant difference between student performance on Form A and B of 
the District Vocabulary Test, descriptive statistics (see Table 4) and analysis of variance (see 
Table 5) include data from both forms combined. 

Table 4 



Descriptive Statistics for Grade 7 District Vocabulary Test 





Group 


n 


M 


SD 


Gender 


Male 


150 


75.28 


17.66 




Female 


152 


74.06 


18.85 


Total 




302 


74.66 


18.25 



AOV finds no difference in student performance on the District Vocabulary Test with regards to 



gender. 
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Table 5 



Analysis of Variance Summary Table for Grade 7 District Vocabulary Test 



Source 


df 


F 


2 


p 


Gender 


1 


.335 


.00 


.56 


Error 


300 


(.033) 







Note. Items in parentheses represent mean square errors. 
*p < .05, **p < .01. 

District Reading Comprehension Test 



There is a statistically significant difference between student performance on the SR 
portion of the four forms, F (3, 317) = 5.87, p < .01. Table 6 presents a comparison of the four 
forms. Students perform at a significantly lower level on Form C {Merrick and the Challenge of 
Denali passage). There is a statistically significant difference between student performance on 
the CR portion of the four forms F (3, 317) = 81.54, p < .001. Students perform at a significantly 
lower level on Form C {Merrick and the Challenge of Denali passage) and at a significantly 
higher level on Form A {Fenner passage). 

Table 6 



Descriptive Statistics for Grade 7 District Reading Comprehension Test 



Form 


n 


SR Mean 


SRSD 


CR Mean 


CKSD 


A 


75 


74.29% 


16.46 


89.17% 


25.28 


B 


83 


71.03% 


16.93 


72.05% 


27.40 


C 


81 


64.73% 


17.80 


20.99% 


34.45 


D 


82 


74.00% 


14.24 


63.78% 


26.14 



For this reason. Form C is separated out from the other three forms for analyses of student 
performance by group in the SR and CR section while form A is separated out from the others 
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analyses of the CR section. Forms B and D are combined for analyses of the CR section. In 
addition, only forms B and D are used for correlational analyses. Table 7 presents descriptive 
statistics for Forms A, B, and D, combined. 

Table 7 



Descriptive Statistics for Grade 7 District Reading Test: SR Forms A, B, and D 





Group 


n 


M 


SD 


Gender 


Male 


108 


73.45 


15.26 




Female 


115 


73.48 


16.07 


Total 




223 


73.46 


15.65 



On Forms A. B, and D combined, there was no significant difference in performance on 
the SR section between different groups of students with regards to gender (see Table 8). 

Table 8 



Analysis of Variance Summary Table for Grade 7 District Reading Test: SR, Forms A, B, and D 



Source 


df 


F 




P 


Gender 


1 


0.00 


.00 


.99 


Error 


82 


(3.16E-02) 







Note. Items in parentheses represent mean square errors. 
*p < .05, **p < .01. 



Table 9 presents descriptive statistics for student performance on the SR portion of Form C of 
the District Reading Comprehension test. 
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Table 9 



Descriptive Statistics for Grade 7 District Reading Test: SR Form C 





Group 


n 


M 


SD 


Gender 


Male 


40 


60.83 


19.12 




Female 


36 


68.78 


15.95 


Total 




76 


64.60 


18.03 



There was not a significant difference between student performance in regard to gender 
F(l, 74) = 3.82, p = .05 on the SR section of the District Reading Test, Form C (see Table 10). 

Table 10 



Analysis of Variance Summary Table for Grade 7 District Reading Test: SR, Form C 



Source 


df 


F 




P 


Gender 


1 


3.82 


.05 


.05 


Error 


74 


(.03) 






Note. Items in parentheses represent mean square errors. 




*p < .05, **p < .01. 








Table 1 1 presents descriptive statistics for the CR section of the District Reading Test, 


Form A. 










Table 11 










Descriptive Statistics for Grade 7 District Reading Test: CR, Form A 




Group 


n 


M 


SD 


Gender 


Male 


40 


26.25 


36.23 




Female 


36 


17.71 


33.99 


Total 




76 


22.20 


35.21 




Reading Analysis 7* Grade - Page 9 



There was not a significant difference between student performance in regard to gender 
F(l, 74) = 1.12, p > .05 on the CR section of the District Reading Test, Form A (see Table 12). 

Table 12 



Analysis of Variance Summary Table for Grade 7 District Reading Test: CR. Form A 



Source 


df 


F 




P 


Gender 


1 


1.12 


.02 


.29 


Error 


74 


(.12) 






Note. Items in parentheses represent mean square errors. 




*p < .05, **p < .01 








Table 13 presents descriptive statistics for the CR section of the District Reading Test, 


Forms B and D. 








Table 13 










Descriptive Statistics for Grade 7 District Reading Test: CR, Forms B and D 


Group 


n 


M 


SD 


Gender 


Male 


79 


73.04 


14.67 




Female 


75 


71.84 


16.73 


Total 




154 


72.46 


15.67 



There was also not a significant difference between student performance in regard to gender F(l, 
152) = 0.23, p > .05 on the CR section of the District Reading Test, Forms B and D (see Table 



14 ). 
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Table 14 



Analysis of Variance Summary Table for Grade 7 District Reading Test: CR, Forms B and D 



Source 


df 


F 


2 


P 


Gender 


1 


0.23 


.00 


.64 


Error 


152 


(.025) 






Note. Items in parentheses represent mean square errors. 




*p < .05, **p < .01 








Table 15 presents descriptive statistics for the CR section of the District Reading Test, Form C. 


Table 15 










Descriptive Statistics for Grade 7 District Reading Test: CR Form C 




Group 


n 


M 


SD 


Gender 


Male 


40 


26.25 


36.23 




Female 


36 


17.71 


33.99 


Total 




76 


22.20 


35.24 



There was not a significant difference between student performance in regard to gender 
F(l, 74) = 1.17, p > .05 on the CR section of the District Reading Test, Form C (see Table 16). 

Table 16 



Analysis of Variance Summary Table for Grade 7 District Reading Test: CR, Form C 



Source 


df 


F 




P 


Gender 


1 


1.17 


.02 


.29 


Error 


74 


(.124) 







Note. Items in parentheses represent mean square errors. 
*p < .05, **p < .01. 
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Correlation of the Four Measures 

Because the SR section of Forms A and C and the CR section of Form C of the District 
Reading Test differ significantly from the other forms, they are excluded from the remaining 
analyses. There is a significant positive correlation between all of the measures, with the highest 
correlation (.43) between the CR section of the District Reading Test and the District ORF (see 
Table 17). 

Table 17 



Correlation Matrix for the Different District Tests 







District 

ORF 


District 

Voc. 


District 

SRRdg 


District 

CRRdg 


District 


Pearson Correlation 


1 


43** 


27** 


43** 


ORF 


Sig. (2-tailed) 




.000 


.000 


.000 




n 


141 


141 


141 


141 


District 


Pearson Correlation 




1 


.26** 


44** 


Voc. 


Sig. (2-tailed) 






.002 


.000 




n 




146 


146 


146 


District SR 


Pearson Correlation 






1 


.30** 


Reading 


Sig. (2-tailed) 








.000 




n 






146 


146 


District CR 


Pearson Correlation 








1 


Reading 


Sig. (2-tailed) 











n 146 



**. Correlation is significant at the .01 level (2-tailed). 
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Discussion 

ORF 

The ORF as it was administered in 2002-03 is moderately correlated with same year 
performance on the District Reading Test (r = .43) and same year performance on the District 
Vocabulary Test (r = .43). Because the ORF is easy to administer and does not require much 
time or training to score, it can continue to be a useful source of information for teachers 
monitoring student growth in reading. 

District Vocabulary Test 

Both forms of the District Vocabulary Test are functioning correctly, although the district 
may want to make the tests more challenging in order to obtain more information from them. 
They currently do not offer as much differentiation as would be possible were they scaled more 
aggressively. 

District Reading Comprehension Test 

The district administered four different forms of the Reading Comprehension Test. Two 
of the reading passages were fiction (Forms A and B) while two were non-fiction (Forms C and 
D). The difference in type of literature (fiction versus non-fiction) did not have a significant 
effect on student performance. All four forms had different numbers of questions and varied 
slightly in length and degree of difficulty on the Flesch-Kinkaid reading scale. Table 18 presents 
comparative information on the four forms. 




Reading Analysis 7* Grade - Page 13 



Table 18 



Comparison of the Four District Reading Comprehension Forms 



Form 


Number 

of 

Words 


Reading 

Level 


Number 
of SR 
Questions 


Mean 

SR 

Score 


#ofCR 

Questions 


Mean CR 
Score 


A 


1569 


6.9 


21 


79% 


4 


94% 


B 


1396 


7.7 


22 


79% 


5 


77% 


C 


1397 


8.2 


21 


69% 


4 


71% 


D 


1396 


8.0 


25 


79% 


5 


68% 



Forms B and D are comparable based on an ANOVA of student performance on the tests. 
There is no significant difference between student scores on these two forms. However, Forms A 
and C are not equivalent to the other two forms; as a result, scores on Forms A and C can not 
reliably be compared to scores on the other three forms. 

The district asked for a recommendation of questions that could be removed to reduce the 
SR section of each form to 15 questions and the CR section of each form to 2 questions. Table 19 
presents recommendations on which items to remove based on an analysis of how the different 



items are functioning. 
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Table 19 



Items for Removal from Grade 7 Reading Test and How Removal Would Affect Scores 



Form 


SR Item #s 
for 

Removal 


New Mean 
SR Score 


SR Score 
Before 
Removal 


CR Item #s 
for 

Removal 


New Mean 
CR Score 


CR Score 
Before 
Removal 


A 


1,2, 3, 6, 9, 
16 


79% 


79% 


23,25 


91% 


94% 


B 


1,5, 8, 12, 
18,21,22 


69% 


79% 


23, 24, 26 


67% 


77% 


C 


3,7, 12, 15, 
20,21 


71% 


69% 


23,25 


68% 


71% 


D 


1,2, 3, 5, 9, 
10, 17, 18, 
23,25 


68% 


79% 


26, 28, 29 


70% 


68% 



Recommendations are based on student performance. Table 20 provides a rationale for each of 
the items recommended for removal. Based on mean student performance, removing the 
suggested SR items brought Forms B, C, and D closer together, but Form A remains easier than 
the others. To make Form A more comparable to the other three Reading Comprehension Test 
forms, more difficult test items need to be written for it. In reading Table 20, an item is 
considered redundant if students performed equally well on that item as they did on another item 
on the same form. The % given in parentheses refers to the percentage of seventh-grade students 
who got that particular item correct. A distracter is referred to as a bad distracter when no 
students selected that particular response; distracters that were selected by no students are noted 
in the Action Needed to Save Item for Question Bank column. See Appendix A for a complete 



table of Item Analysis for the SR section of the District Reading Test. 
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Table 20 



Rationale for Items Suggested for Removal from Grade 7 District Reading Test 



Form 


Item 


Rationale for Removal 


Action Needed to Save Item for Question Bank 


A 


1 


Too easy (96%) 


Re-write question 


A 


2 


Redundant with 5, 14, 18, 20, 
21 and one bad distracter. 


Re-write Distracter B 


A 


3 


Two bad distracters 


Re-write Distracters B and D 


A 


6 


Redundant with 13 and 15. 


OK to use as is in place of 13 or 15 


A 


9 


Too easy (97%) and distracter B 
is bad 


Re-write question and Distracter D 


A 


10 


Redundant with 6, 13, and 15 


OK to use as is in place of 6, 13 or 15 


A 


16 


Redundant with 2, 5, 14, 18, 
and 21 and less a test of reading 
than knowledge of hospitals 


Re-write question 


A 


23 


Too easy (94%) 


Re-write question to make more challenging 


A 


25 


Too easy (96%) 


Re-write question to make more challenging 


B 


1 


Too easy (97%) and distracter 
A is bad 


Re-write question and Distracter A 


B 


5 


Redundant with 13 and one bad 
distracter 


Re-write Distracter C 


B 


8 


Redundant with 12 and 18 


OK to use as is in place of 12 or 18 


B 


12 


Redundant with 8 and 18 and 
one bad distracter 


Re-write Distracter A 


B 


18 


Redundant with 8 and 12 


OK to use as is in place of 8 or 12 


B 


21 


Redundant with 9 and two bad 
distracters 


Re-write Distracters A and D 


B 


22 


Too easy (97%) and two bad 
distracters 


Re-write to make question more challenging, 
and re-write Distracters A and D 


B 


23 


Too easy (83%) 


Re-write question to make more challenging 
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B 


24 


Too easy (80%) 


Re-write question to make more challenging 


B 


26 


Too easy (88%) 


Re-write question to make more challenging 


C 


3 


Redundant with 1 and 12 and 
one bad distracter 


Re-write Distracter D 


C 


7 


Redundant with 2 and one bad 
distracter 


Re-write Distracter A 


C 


12 


Redundant with 1 and 3 


OK to use as is in place of 1 or 3 


C 


15 


Redundant with 18 


OK to use as is in place of 18 


C 


20 


Too hard (12%) 


Re-write question 


C 


21 


Redundant with 17 


OK to use as is in place of 17 


C 


23 


Too easy (77%) 


Re-write question to make more challenging 


C 


25 


Too easy (81%) 


Re-write question to make more challenging 


D 


1 


Redundant with 5, 10, 15, and 
19 and two bad distracters 


Re-write Distracters B and D 


D 


2 


Too easy (95%), redundant with 
3, 12, and 18 and one bad 
distracter 


Re-write to make more challenging, and re- 
write Distracter C 


D 


3 


Too easy (95%), redundant with 
2, 12, and 18, and two bad 
distracters 


Re-write to make more challenging, and re- 
write Distracters B and C 


D 


5 


Redundant with 1, 10, 15, and 
19 and one bad distracter 


Re-write Distracter C 


D 


9 


Redundant with 7 and 16 


OK to use as is in place of 7 or 16 


D 


10 


Redundant with 1,5, 15, and 19 
and one bad distracter 


Re-write Distracter D 


D 


17 


Too easy (94%) and one bad 
distracter 


Re-write question to make more challenging, 
and re-write Distracter D 


D 


18 


Too easy (95%), redundant with 
2, 3, and 12, and two bad 
distracters 


Re-write question to make more challenging, 
and re-write Distracters A and D 
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D 


23 


Redundant with 8 and one bad 
distracter 


Re-write Distracter C 


D 


25 


Redundant with 1 1 and one bad 
distracter 


Re-write Distracter D 


D 


26 


Too easy (87%) 


Re-write question to make more challenging 


D 


28 


Too easy (81%) 


Re-write question to make more challenging 


D 


29 


Too hard (34%) 


Re-write question to make less challenging 



The district’s current reading assessment kit is a commendable model. It can offer 
insights into strengths of particular programs, schools, and teachers and provides school 
personnel with information that can help them measure their progress towards promoting reading 
proficiency for all students. 
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Appendix A 

Correlat 

ion 



between 

student 

score 







%of 










Mean 


Mean 


Mean 


Mean 


and 






students 


%of 


%of 


%of 


%of 


score of 


score of 


score of 


score of 


selectio 






who got 


students 


students 


students 


students 


students 


students 


students 


students 


n of 






item 


selecting 


selecting 


selecting 


selecting 


selecting selecting 


selecting 


selecting 


right 


Item 


Form 


correct 


Option A 


Option B 


Option C 


Option D 


Option A Option B 


Option C 


Option D 


answer 


1 


A 


96 


01 


01 


96 


01 


0.38 


0.24 


0.83 


0.38 


0.57 


2 


A 


85 


01 


00 


14 


85 


0.76 


0.00 


0.69 


0.83 


0.30 


3 


A 


70 


70 


00 


30 


00 


0.82 


0.00 


0.80 


0.00 


0.06 


4 


A 


76 


05 


16 


76 


03 


0.65 


0.78 


0.84 


0.60 


0.27 


5 


A 


84 


84 


13 


00 


04 


0.84 


0.68 


0.00 


0.52 


0.44 


6 


A 


90 


06 


90 


03 


01 


0.78 


0.82 


0.60 


0.76 


0.16 


7 


A 


62 


28 


06 


03 


61 


0.76 


0.59 


0.79 


0.86 


0.36 


8 


A 


80 


80 


05 


09 


06 


0.85 


0.55 


0.69 


0.71 


0.45 


9 


A 


97 


01 


00 


01 


96 


0.62 


0.00 


0.67 


0.82 


0.18 


10 


A 


90 


04 


89 


03 


04 


0.68 


0.83 


0.81 


0.71 


0.20 


11 


A 


71 


70 


01 


11 


16 


0.86 


0.90 


0.63 


0.74 


0.46 


12 


A 


72 


14 


71 


05 


09 


0.68 


0.86 


0.76 


0.71 


0.45 


13 


A 


90 


01 


06 


03 


87 


0.71 


0.67 


0.33 


0.85 


0.53 


14 


A 


86 


10 


03 


84 


01 


0.65 


0.55 


0.85 


0.86 


0.46 


15 


A 


90 


08 


01 


87 


01 


0.53 


0.38 


0.86 


0.81 


0.62 


16 


A 


83 


80 


14 


03 


00 


0.86 


0.75 


0.33 


0.00 


0.45 


17 


A 


89 


86 


08 


01 


01 


0.86 


0.69 


0.43 


0.29 


0.53 


18 


A 


86 


05 


82 


01 


08 


0.73 


0.85 


0.38 


0.75 


0.36 


19 


A 


74 


01 


71 


05 


19 


0.76 


0.86 


0.73 


0.74 


0.37 


20 


A 


84 


03 


04 


09 


81 


0.60 


0.60 


0.70 


0.86 


0.52 


21 


A 


84 


03 


08 


80 


05 


0.48 


0.70 


0.86 


0.69 


0.52 
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